Identifying "aboutness topics": two annotation experiments

This paper deals with the annotation of “aboutness topic” (also known as “sentence topic”) in naturally occurring data. We report on two annotation experiments involving German newspaper texts: In experiment 1, based on the annotation guidelines by Götze et al. (2007), two expert annotators had to select the aboutness topic from among a small number of pre-defined choices, for a total of 588 sentence tokens. Although the results are disappointing in terms of inter-annotator agreement (with Fleiss’ κ between .19 and .57, depending on sentence category), they allowed us to identify and analyze cases that resist easy classification. Next, the original guidelines were tentatively modified, capturing insights from experiment 1. In a second experiment, based on the revised guidelines and following several training sessions, four raters annotated a total of 56 sentences, deciding for each NP whether or not it is an aboutness topic. Overall, the results are still disappointing from an inter-rater agreement perspective, but again, a closer look at the problematic cases is instructive as to the weaknesses of the current operationalizations of information structural notions and, perhaps most importantly, it reveals a number of basic theoretical questions that still await satisfactory answers.